Repository navigation
Conversation
sezruby
force-pushed
the
fix/batch-ivf-segment-ownership
branch
from
October 1, 2026 04:19
7274544 to
8dce5b7
Compare
…ce-format#9930) `RepDefUnraveler::decimate` ended in `todo!("Not yet supported FSL<...List<...>>")` whenever the fixed-size-list layer was reached with repetition levels present. A `FixedSizeList<Struct<List<Int32>>>` column writes fine at the stable file version, and every read of it then panicked inside a decode task, so the caller got `Wrapped error: task 23 panicked with message "not yet implemented: Not yet supported FSL<...List<...>>"` rather than a typed error. It now returns `Error::not_supported`, which `rust/AGENTS.md` asks for on unimplemented paths. The zero-dimension arm of the same function already returns a typed error; lance-format#8618 made that one `invalid_input`, which fits there because a zero dimension is bad caller data rather than a missing feature. Reads of such a column still fail, but with a descriptive error instead of an unwound panic. lance-format#9926 has the per-version table. V2_1 rejects the schema at write time, so the versions that accept it have no write-side gate. Whether a given column hits this also depends on which struct child carries the list, because `RepDefStructDecodeTask` keeps child 0's repdef and discards the dense siblings. That is why `test_fsl_struct_random::case_3_struct_with_list` passes today: its fixture puts `id` first. Fixes lance-format#9926. ## How was this patch tested `test_repdef_fsl_over_list_is_not_supported` builds the repdef layers the writer emits for `FixedSizeList<Struct<List<Int32>>>` (fixed-size-list, struct, list offsets, leaf validity), consumes the inner layers in decode order, and asserts the fixed-size-list layer returns `NotSupported`. Restoring the `todo!()` makes it fail on the panic instead. A scratch probe over all four selectable file versions, not part of the patch, moved the V2_2 and V2_3 reads from the wrapped panic message to `Not supported: Reading a column that combines a fixed-size-list with a list-like layer ...`. V2_0 and V2_1 still reject the write. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
`modifies_same_metadata` compared `schema_metadata_updates` with a blanket `is_some() && is_some()` check, while the table-metadata comparison a few lines above routes through `update_maps_conflict`, which conflicts only on a shared key or when either side sets `replace`. `UpdateMetadataBuilder` produces merge-mode maps by default, so two concurrent commits that each set a different schema-metadata key were rejected with a non-retryable `IncompatibleTransaction`. lance-format#9972 has the repro and the error the second writer gets. Route schema metadata through `update_maps_conflict` as well. Either side setting `replace` still conflicts, which keeps the behavior the whole-write path depends on. Fixes lance-format#9972. ## How was this patch tested `test_schema_metadata_conflict_is_per_key` mirrors the neighbouring `test_table_metadata_conflicts_on_same_key`: the same key conflicts, different keys do not, and a replace-mode map conflicts. Restoring the blanket check fails the disjoint-key assertion. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…ion (lance-format#9976) `is_field_binding_fully_rewritten` checked materialization with a plain `fields.contains(&field_id)`, while the sibling `file_materializes_field`, used for the same concept elsewhere in this validation, recurses into a struct's children. STABLE files record only leaf ids, so a `Merge` that changes a struct's own binding while rewriting every fragment was rejected, and the prior-file-replaced check below it was vacuous for a struct id. lance-format#9973 has the error text and the public commit surface that reaches it. Both sides now go through `file_materializes_field`, with the field each side is actually about: a new file has to carry data for the proposed binding, and the files that have to be gone are the ones carrying the prior binding. Passing both fields in also drops the schema lookup the first version used, which searched only the top-level fields and left a nested struct rejected. Fixes lance-format#9973. ## How was this patch tested `test_merge_struct_binding_fully_rewritten_stable_files` has two cases, a top-level struct and a nested one. Each flips the nullability of the struct whose binding changes and rewrites the single fragment with a V2_2 file that lists only leaf ids. Both fail against main: the top-level case on the plain `contains`, the nested case on a lookup that only sees top-level fields. `test_merge_struct_binding_checks_the_proposed_field` covers the other direction: the struct keeps its id and flips to nullable while its child becomes a different field, and the replacement file carries only the prior child. Matching new files against the prior field accepts that merge even though the file holds no data for the proposed binding; matching against the proposed field rejects it. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…ance-format#10211) fixed lance-format#10210 schema_fragments_modern_valid only checked that each data file had at least one field, so committing an Append whose fragments omit a column declared non-nullable succeeded. That column is then physically absent and read back as all-null, violating the schema -- silent since lance-format#9562 (previously an Arrow non-null panic on read). It is reachable through the low-level write_fragments + LanceOperation.Append + commit path, which never runs check_compatible. Require every non-nullable field whose ancestors are all non-nullable to be present in each fragment's data files, mirroring the v1 schema_fragments_legacy_valid check scoped to non-nullable fields. Nullable fields (and anything under a nullable ancestor) may still be omitted per the subschema-insert contract (lance-format#3016 / lance-format#3041). Change-Id: I5a9a018272ac0b5f5941c2d6ff3e705bdcf647f7 --------- Co-authored-by: wangpeipei05 <wangpeipei05@baidu.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…at#10080) Closes lance-format#10078 ## Motivation pylance builds with the `geo` feature, which is a default feature of the `lance` crate, so its spatial predicates run through geodatafusion. When the constant geometry is the first argument, geodatafusion's `relate_impl` computes the DE-9IM matrix for (B, A) but reads it as (A, B). As a result, `ST_Contains(<region>, geom)` returns false for the points inside the region. `delete("NOT ST_Contains(<region>, geom)")` deletes every row, including the ones inside the region. A NULL constant geometry panics the scan. lancedb/lancedb#4555 is about to turn on `geo` in the LanceDB Python and TypeScript SDKs, so more users will hit this. The bug is in geodatafusion 0.5.0, which `main` pins, and in 0.6.0, which lance-format#9979 moves to. The upstream fix, datafusion-contrib/geodatafusion#111, is still open. This PR fixes the bug in Lance, so it works with either version and doesn't wait for a geodatafusion release. ## What changes `lance-geo` now registers its own versions of the ten relate predicates: `st_contains`, `st_within`, `st_covers`, `st_coveredby`, `st_crosses`, `st_disjoint`, `st_equals`, `st_intersects`, `st_overlaps` and `st_touches`. These replace geodatafusion's registrations of the same names. They live in `rust/lance-geo/src/udf.rs`. That file was already on `main`, but no `mod udf` declaration compiled it, and `geodatafusion::register` already registers everything it listed, so this PR replaces its contents. Each predicate wraps geodatafusion's implementation and changes three things: - **Scalar first, array second.** The call is evaluated as the converse predicate with the arguments swapped: `ST_Contains(c, x)` = `ST_Within(x, c)` and `ST_Covers(c, x)` = `ST_CoveredBy(x, c)`. The symmetric predicates are their own converse. This uses geodatafusion's (array, scalar) path, which is correct and keeps its prepared-geometry fast path. - **NULL geometry.** A NULL scalar geometry in either position returns NULL instead of panicking. - **Index use.** `ScalarUDFImpl::simplify` rewrites `f(<literal>, x)` to `converse(f)(x, <literal>)`. The R-tree matcher (`visit_scalar_fn`) only takes the indexed column from the first argument, so constant-first predicates never used the index before this change; now they do. The rewrite also covers second arguments that are not columns, such as `ST_Contains(<region>, ST_Point(x, y))`. ## Design The fix is at the UDF level rather than in the planner, because that is the one place every evaluation path shares. All of these paths resolve the functions by name from a registry that `lance_geo::register_functions` fills: - the scanner filter - `delete`, `update` and `merge_insert` conditions - Substrait filters - `Dataset::sql` - downstream sessions built with `lance_datafusion::udf::register_functions` or `new_session_context`, which is how LanceDB sets up its sessions The argument-order rewrite is in the UDF's `simplify` hook rather than in a Lance-specific planner pass. DataFusion's `ExprSimplifier` runs that hook both for `Planner::optimize_expr` and for the DataFusion optimizer behind `Dataset::sql`. Constant folding runs first, so `ST_GeomFromText('...')` has already become a literal when the rewrite sees it. The invoke-time swap is still needed for expressions that reach execution without being simplified, such as a raw `Expr` passed to `Scanner::filter_expr`. There are no `Cargo.toml` or `Cargo.lock` changes. ## Affected predicates (verified) I compared all ten relate predicates, in both argument orders, against `geo::Relate`. The test geometries were points, lines and polygons that lie inside a square, on its boundary, or outside it, or that overlap, cross, equal, touch or enclose it: 156 pairs per predicate. This used a temporary `geo` dev-dependency that is not part of this PR. | Predicate | geodatafusion 0.5.0, scalar first | With this PR | | --- | --- | --- | | `ST_Contains` | 30 of 156 pairs wrong | all correct | | `ST_Within` | 30 of 156 pairs wrong | all correct | | `ST_Covers` | 44 of 156 pairs wrong | all correct | | `ST_CoveredBy` | 44 of 156 pairs wrong | all correct | | `ST_Crosses`, `ST_Overlaps` (including mixed dimensions), `ST_Touches`, `ST_Equals`, `ST_Intersects`, `ST_Disjoint` | all correct | all correct | Every (array, scalar) result was already correct before this change. The NULL panic affects all ten predicates, with the NULL in either position. ## Testing `lance-geo` unit tests, run with `--features geo`: - Every predicate, with the constant in either position, against geodatafusion's (array, array) evaluation. That path relates each pair of rows with `geo::Relate` in argument order. - The constant-first result against the column-first converse. - Known answers for the asymmetric predicates. - NULL constants, typed and untyped, in both positions, on both a non-empty and an empty batch. - The `simplify` rewrite. - That `register_functions` installs the wrappers. `lance` dataset tests in `dataset_geo.rs`. Each runs with and without an R-tree, on a dataset with several fragments that includes a NULL geometry: - **Reads**, through the scanner filter, `Dataset::sql`, and a `SessionContext` set up with `register_functions` and `LanceTableProvider`. The constant-first result must equal the column-first converse and the expected rows. When the column is indexed, the plan for the constant-first form must show `ScalarIndexQuery`. - **Wrong-row mutation regressions.** `delete` with `NOT ST_Contains`, `ST_Contains`, `NOT ST_Covers`, `NOT ST_Within` and `NOT ST_CoveredBy`, and `update` with `ST_Contains`, `NOT ST_Contains`, `ST_Covers` and `NOT ST_Within`. Each test asserts exactly which rows survive or change. - **`merge_insert`** with `WhenNotMatchedBySource::delete_if("ST_Contains(<region>, geom)")`. - **NULL constant geometry**, in reads and in `delete`. Python tests in `test_geo.py`: constant-first reads, `update` and `delete`, each with and without an R-tree, plus the NULL constant. All of the new Rust and Python tests fail when the wrappers are not registered, and pass when they are. With only the `simplify` rewrite disabled, results stay correct but the index-use assertions fail. Commands run: - `cargo fmt --all -- --check` - `cargo clippy --profile ci --locked -p lance-geo --features geo --all-targets -- -D warnings` - `cargo clippy --profile ci --locked --all --tests --benches -- -D warnings` - `cargo test --profile ci -p lance-geo --features geo` - `cargo test --profile ci -p lance --lib -- geo rtree st_intersects` - `cargo test --profile ci -p lance-index --features geo --lib -- rtree geo` - From `python/`, with the `geo` extra installed: `uv run make lint` and `uv run pytest python/tests/test_geo.py`. The new tests pass. `test_staged_rtree_after_rewrite_columns` fails with the same assertion on a build without this change. CI installs only the `tests` and `ray` extras, so it skips `test_geo.py` (`importorskip("geoarrow.rust.core")`). The Rust tests cover the same behavior. ## Follow-up Once Lance pins a geodatafusion release that includes datafusion-contrib/geodatafusion#111, remove the argument swap and NULL handling in `RelateUdf::invoke_with_args` in `udf.rs`. The `simplify` rewrite does not depend on the upstream bug and keeps constant-first predicates indexable, so it should stay. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…fix (lance-format#10228) `extract_like_leading_prefix` now reads a `LIKE` pattern the way the executor does. Fixes lance-format#10226. The extractor makes a single left-to-right pass. An escape consumes whatever character follows it, as arrow's LIKE kernel does, so `a\b%` gives the prefix `ab` instead of `a\b`, and the wildcard check uses the same pass, so `\\\%` is the literal `\%` rather than a prefix on it. Before, a BTREE answered `LIKE 'a\b%'` with rows starting with `a\b` and no recheck, and a multi-fragment ZONEMAP pruned the fragments holding the matching rows. A non-backslash `ESCAPE` character now gets no index acceleration: DataFusion's physical planner refuses it, and an index query that replaces the predicate would answer it instead. So `LIKE 'a!%%' ESCAPE '!'` on a BTREE-indexed column, which used to return rows, now fails the way it already did without an index. `starts_with` is unaffected: DataFusion rewrites it into a pattern that only escapes `\`, `%` and `_`, which both versions read the same way. ## How was this patch tested `test_extract_like_leading_prefix` now expects `foo\bar%` to give `foobar`, where it asserted `foo\bar` before, and adds `\\\%`, `\\\%%` and an `ESCAPE '!'` case. `test_like_expression_parsing` plans `color LIKE 'a\b%'` as `LikePrefix("ab")` with no refine and leaves `ESCAPE '!'` unaccelerated. `test_like_prefix_escapes_match_unindexed_scan` runs `LIKE 'a\b%'` and `LIKE '\\\%'` through a BTREE and a three-fragment ZONEMAP and compares the rows with an unindexed scan; both cases fail without the change.
…ance-format#10175) ## Problem A merge insert that deletes matched rows, inserts nothing, and deletes target rows not matched by the source (`when_not_matched_by_source` = `Delete` or `DeleteIf`) fails with an internal error whenever the source carries a dataset column other than the key: ``` Dataset field "k" missing from merge insert input schema — this indicates a logical plan pruning bug ``` Example: target `(id, k, emb)`, source `(id, k)`, `on = "id"`. The same merge with a source of only `id` works. Cause: two checks disagree on what a delete-only merge is. - `MergeInsertWriteNode::necessary_children_exprs` pruned the source to its key columns for any `Delete` without inserts. - `MergeInsertPlanner` only picks `DeleteOnlyMergeInsertExec` when `when_not_matched_by_source` is also `Keep`. - Otherwise it picks `FullSchemaMergeInsertExec`. That exec looks up every dataset column by name and finds `k` pruned away. ## Fix In `necessary_children_exprs` (`merge_insert/logical_plan.rs`): - Prune the source to its key columns only when `MergeInsertPlanner::is_delete_only` holds, the same check that picks the exec. - Otherwise keep only the source columns that exist in the dataset (plus the source sentinel), not every source column. - No exec reads a source column the dataset lacks. - Such columns can only reach this plan in merges that write nothing, because updates and inserts require the source to be the full dataset schema or a subset of it. - Dropping them keeps the earlier pruning for a source like `(id, extra)` and stops `extra` from riding through the join. ## Effect Benchmark setup: - Target: 1M rows, 6 columns, including a 32-d vector and a struct. - Source: 510K rows. - Release build, one merge per process, main and this PR interleaved, medi | Case | main | this PR | |---|---|---| | Matched `Delete` + not-matched-by-source `Delete`/`DeleteIf`, source car| error | works; deletes the same rows as a key-only source | | Matched `DoNothing` + not-matched-by-source `DeleteIf`, source carries a column not in the dataset | 627 ms, +358 MB peak RSS | 409 ms, +162 MB peak RSS | | 9 other combinations (delete-only, upsert, delete + insert, ...) | — | identical physical plans and results; timings within noise |
…ns (lance-format#10135) A projection expression that uses a system column inside a larger expression now loads that column. Fixes lance-format#10129. `ProjectionBuilder::add_column` set the flags that request `_rowid`, `_rowaddr`, `_rowoffset`, `_row_last_updated_at_version` and `_row_created_at_version` only when the whole expression was that column, so `columns={"next": "_rowid + 1"}` failed with `No field named _rowid`. The check now runs over every column the expression references, which also covers the bare-column case it replaces. This applies to `Scanner::project_with_transform` and to `take` with SQL projections. ## How was this patch tested `test_scan_with_wildcard` projects each of the five system columns plus one and checks the output schema, and checks that `_rowoffset + 1` yields 1 to 100. It fails without the change.
…lance-format#10134) `is_substrait_compatible` now recurses into `LargeList` and `Map` the way it already does into `List`. Fixes lance-format#10128. DataFusion's producer encodes the whole schema, not only the columns a filter references, and has no Substrait type for `FixedSizeList`. `prune_schema_for_substrait` drops such columns before encoding, but a `large_list<fixed_size_list>` or `map<_, fixed_size_list>` column was reported compatible and kept, so encoding any filter on that dataset failed with `Unsupported cast type: ... FixedSizeList`. That is the encoding `filtered_read_exec_to_proto` does for a filtered read. The decoder takes the pruned schema from the message, so it needs no change. This complements lance-format#9057, which checks the columns a filter references with `is_substrait_compatible` before pruning; with this change that check also catches a filter on a `large_list<fixed_size_list>` column. ## How was this patch tested `test_prune_schema_for_substrait_nested_list_types` builds a schema with a plain large list, a plain map, and a large list and a map holding fixed-size lists. It asserts that only the two holding fixed-size lists are dropped and round-trips a filter on another column through the pruned schema. It fails without the change.
…ormat#10133) `contains_tokens(text, query)` read only the first row of `query` and applied it to every row. Fixes lance-format#10127. The UDF now passes `make_scalar_function` hints so a literal query stays one value, tokenized once per batch, and a column query is read row by row. A NULL text or NULL query gives NULL. Before, a NULL query read as `""`, an empty token list that every text contains, so `contains_tokens(text, NULL)` matched every non-null row; a column query applied its first row's value to all rows. The FTS index only takes a non-null string literal, so both cases ran through the UDF with or without an index. Results for a non-null literal query are unchanged. ## How was this patch tested `test_contains_tokens_null_query_matches_nothing` passes a NULL literal, and `test_contains_tokens_reads_a_column_query_per_row` passes a column whose first value differs from the second, so reading only the first row gets the second row wrong. `test_sql_contains_tokens` adds `contains_tokens(text, NULL)` through SQL, before any index is built and with a suitable FTS index, and asserts no rows. All three fail without the change.
…sts, narrow under NOT (lance-format#10131) Three fixes to how `lance-datafusion` turns filter and expression strings into DataFusion expressions. Fixes lance-format#10125. `Planner::value` returns an `InvalidInput` error for a SQL value it has no mapping for, instead of reaching `todo!()`. Placeholders (`:id`, `$1`, `?`) and national strings (`N'abc'`) used to panic, including inside an array literal; from Python that panic is a `PanicException`, which `except Exception` does not catch. `is_valid()` now requires exactly one argument: with none it panicked on `args[0]`, and extra arguments were dropped. `parse_sql_expr` rejects input that parses to more than one `SELECT` item. It used to keep the first item and drop the rest, so `update({"x": "coalesce(x), 0"})` committed `x = coalesce(x)`. Every caller of `Planner::parse_expr` (`update`, `add_columns`, `columns={...}` projections) passes one expression per string. `resolve_expr` recurses into `NOT` and `IS [NOT] TRUE/FALSE`, so the literal inside is narrowed to the column type the same way it is outside. Without that, on a `Float32` column `NOT (f = 0.1)` compares `CAST(f AS Float64)` with `0.1`, and the row holding `0.1f32` matches both `f = 0.1` and `NOT (f = 0.1)`. One consequence is that a literal `resolve_expr` already refuses at the top level is now refused under these wrappers as well: `NOT (i = 1.5)` on an `Int32` column used to plan through DataFusion's own coercion and now returns the same error as `i = 1.5`. ## How was this patch tested `test_parse_filter_rejects_unsupported_literals` covers each placeholder form, a national string and one inside an array literal, and asserts the variant and message. `test_parse_filter_checks_is_valid_arity` covers zero and two arguments. `test_parse_expr_rejects_multiple_expressions` uses the misplaced-parenthesis input, and `test_update_validation` passes the same input through `UpdateBuilder::set`. `test_parse_filter_narrows_under_not_and_is` evaluates `=`, `>`, `BETWEEN` and `IN` on a `Float32` column and asserts that `NOT (...)`, `IS FALSE` and `IS NOT TRUE` return the complement, and `IS TRUE` and `IS NOT FALSE` the same rows. Each of these fails without the change.
…s its field (lance-format#10123) Closes lance-format#9724. A `DataOverlay` staged at version N still committed after a concurrent `Project` at N+1 dropped the field it overlays. `check_data_overlay_txn` treated every `Project` as compatible, and the overlay build arm doesn't look at the schema, so the overlay landed covering only fields that no longer exist. lance-format#9314 handles the other order, where `Project` prunes overlays that are already committed. This makes `Project` a retryable conflict for an overlay when the projected schema no longer contains one of the overlay's fields, which is the first option the issue gives. A projection that keeps all of the overlay's fields (a rename, or dropping some other column) is still compatible. Testing: `test_data_overlay_conflicts` gained two `Project` cases, one keeping the overlaid field (compatible) and one dropping it (retryable). The dropping case fails on main. `cargo test -p lance --lib io::commit::conflict_resolver` and `cargo test -p lance --lib overlay` pass, `cargo fmt --all` is clean, and `cargo clippy -p lance --lib --tests -- -D warnings` passes; I didn't run the full-workspace clippy. lance-format#9220 leaves the `(DataOverlay, Project)` cell out of its conflict matrix because of this bug; with this change it should be able to go back in. I used an AI coding assistant to help write this change and the test; I reviewed the diff and ran the checks above.
…e-format#10161) ## Problem Every node of a MemTable's HNSW graph keeps, per level, a published neighbor list behind an `ArcSwap`. Searchers read level 0 from the graph's packed level-0 snapshot (`packed_level0`), which `publish_dirty` rebuilds before any new row becomes visible, so a node's own published level-0 list is never read for a node a search can reach. It is still expensive to free. Dropping an `ArcSwap` walks arc-swap's per-thread debt list, so freeing a graph drops one per node per level, and the graph has a node for every id up to its capacity. A stack sample of a server process freeing a ~414k-row graph at a 1M capacity shows `drop_glue::<HnswGraph>` → `drop_glue::<Vec<LevelLinks>>` → `arc_swap::debt::Debt::pay_all` for about 7 s of CPU. The frozen-MemTable grace sweep frees an evicted generation while holding the shard writer lock, so puts to that shard wait for the whole free. ## Fix - `LevelLinks::published` is `None` on level 0. Upper levels, which about one node in `m` reaches, still publish per node. - `LevelLinks::linked_bytes` no longer charges a per-node level-0 list; level 0 is charged in the packed snapshot. - `mem_wal_hnsw_bench` prints the graph's resident bytes and drop time (`lifecycle` line). ## Measurements `mem_wal_hnsw_bench`, `release-with-debug`, dim 64, m 12, ef_construction 64, 1,000-row batches, 18 threads, Apple M-series; mean of two runs. | rows | capacity | drop, before → after | resident MB, before → after | |---|---|---|---| | 100k | 100k | 0.13 s → 0.024 s | 55.5 → 41.9 | | 100k | 1M | 1.11 s → 0.13 s | 142.6 → 136.9 | | 414k | 1M | 1.17 s → 0.14 s | 286.2 → 235.3 | Recall@10 is unchanged within run-to-run variance. `Option<ArcSwap<_>>` adds 8 bytes to every level header (`ArcSwap` has no niche), about 8.7 MB per million reserved nodes on a graph with no rows; each inserted node saves its level-0 list and `Arc` block (about 128 bytes at m 12), so only very sparsely filled graphs grow. lance-format#10162 allocates slots only for rows that arrive, which removes the fixed part. The grace sweep still frees an evicted generation under the shard writer lock; this change makes that free cheap rather than moving it. ## Tests - `test_parallel_insert_searches_visible_graph` now also checks that no node keeps a level-0 `ArcSwap` and that upper levels still do; it fails with the per-node level-0 list restored. - `test_lance_hnsw_batch_edges_stay_inside_the_exported_prefix`, which reads the graph while a writer inserts, now also checks, in the order a search reads, that every visible node is in the packed level-0 snapshot. It fails if the snapshot is not rebuilt, and if the visible length is published before the snapshot. - `cargo nextest run -p lance --lib -E 'test(/dataset::mem_wal/) | test(/hnsw/)'`: 1052 passed. - `cargo fmt --all` and `cargo clippy -p lance --tests --benches -- -D warnings`: clean. lance-format#10162 changes the same file (node slots grow with rows); whichever merges second needs a one-line update to that test's node iteration.
…ormat#9964) Hierarchical k-means trained each node in `grow_proportional_subtree` on a prefix of its rows, assuming the input was already shuffled. In the non-nullable vector sampling path, [`generate_random_indices`](https://github.com/lance-format/lance/blob/e483590d6df5b767e190913eeb35aa0592614a0a/rust/lance/src/index/vector/utils.rs#L1099-L1119) selects random row positions and then sorts them with `indices.sort_unstable()`. [`sample_fsl_uniform`](https://github.com/lance-format/lance/blob/e483590d6df5b767e190913eeb35aa0592614a0a/rust/lance/src/index/vector/utils.rs#L879-L935) reads those positions through `dataset.take()` in chunks, preserves chunk order with `.buffered(batch_readahead)`, and concatenates the vectors without shuffling them. The full sample is uniformly selected, but its order follows source-row positions; taking a prefix of it is therefore not uniform subsampling. When similar vectors are ingested together—for example, batches from the same source or category—the prefix overrepresents early batches. This biases the hierarchy's splits and centroids, which can worsen IVF partition skew and contribute to empty partitions. Subsample uniformly without replacement at each node, using the existing row-index buffer and node-derived seed. All node rows still participate in assignment, centroid quotas, and leaf means. The upstream sorted reads are preserved. Add regressions for ordered groups and rows appended when merging zero-quota children. Validation: - `cargo test --profile ci --locked -p lance-index vector::kmeans::tests` (38 passed; one existing benchmark ignored) - `cargo fmt --all` - `cargo clippy --profile ci --locked --all --tests --benches -- -D warnings` - `git diff --check`
…ance-format#9963) Packed-struct encoding accepts two child kinds it cannot encode, and both fail destructively. lance-format#9962 has the details and the measured per-version behavior. The short version: an all-null or dictionary-encoded child panics inside `DataBlock::from_arrays`, and a Boolean child writes a column whose values were never stored, or panics the writer once a page reaches 4096 rows. Two changes, one per hole. `StructDataBlock::compute_stat` summed `MaxLength` over every child with `expect_single_stat`, which panics for a child that carries no such statistic. It now sums into an `Option<u64>` and leaves the statistic out when any child has no width to contribute, because an unknown width is not a zero width. `FixedWidthDataBlock::compute_stat` always inserts `MaxLength`, so the sum is absent exactly when some child is not a fixed-width block, which is the case `try_fixed_packed_struct_miniblock` already refuses with a typed error. The only reader of a struct block's `MaxLength` is `is_narrow`, which already treats it as optional and falls back to full-zip. `try_fixed_packed_struct_miniblock` now also rejects a child whose `bits_per_value` is not a positive multiple of 8. `has_variable_width_child` only looks at the block kind, so a Boolean child of one bit passed it, and the packing then wrote `bits_per_value / 8 = 0` bytes for that child while counting its bit in the declared row width. The full-zip packing already refuses a child that is not byte-aligned and the decoder requires a positive width, so this is the mini-block path catching up. Fixes lance-format#9962. ## How was this patch tested `test_struct_stat_without_child_max_length` builds a struct with one all-null child, asserts the block keeps `DataSize` but has no `MaxLength`, and asserts `try_fixed_packed_struct_miniblock` returns a typed error for it. Restoring the `expect_single_stat` sum makes it panic instead. `fixed_packed_struct_miniblock_rejects_sub_byte_child` asserts the typed rejection for a packed `{Boolean, Int32}` struct, naming the offending width. Without the new check the function returns an encoder and the test fails. The decoder side of the same contract is already pinned by a case in `mini_block_descriptor_errors_are_owned_by_codecs`. End to end, with a scratch probe over V2_1, V2_2 and V2_3, a packed `{Boolean, Int32}` column went from "write succeeds, read fails" at 3 rows and a writer panic at 4096 rows to the same typed error in all six cases. A packed struct with an all-null or dictionary child went from a panic out of `write_batch` to `Not supported: Per-value compression not yet supported for block type AllNull` on V2_2 and V2_3, `Not supported: Variable packed struct encoding is not enabled by the selected file format` on V2_1, and `Invalid user input: Packed struct encoder currently only implemented for fixed-width data blocks` on V2_0.
Closes lance-format#9955 Hi team, in the current implementation, it reads to me, even when a `Binary` or `Utf8` field explicitly requests lz4, Lance's variable-width per-value encoding path can use Zstd instead. Simply honoring `lz4` is not enough: full-zip stores no payload for null values, and the LZ4 reader [fails on that empty payload](https://github.com/lance-format/lance/blob/0d42b38f4c8a8425fe479cec06676ad93d781b7b/rust/lance-encoding/src/encodings/physical/block.rs#L411-L415). Released readers (including v13) have the same check, so stable-format files with nullable LZ4 full-zip pages would be unreadable by them. ## What this PR does - **Reader**: `CompressedBufferEncoder::per_value_decompress` now skips empty slots instead of passing them to the decompressor. This is safe for every codec: LZ4 and Zstd length-prefix every valid value (including empty strings), so an empty slot can only be a null. - **Writer**: `try_variable_width_per_value` takes a new `PerValueGeneralCompression` argument that controls which explicitly requested codecs the per-value path honors. - Stable 2.1 / 2.2 pass `ZstdOnly`, which keeps the existing selection: only an explicit `zstd` is honored, and other requests fall back to the default compressor. Stable output is unchanged, so released readers keep reading it. - Unstable 2.3 passes `Any`, which honors the requested codec, including `lz4`. - `fsst` keeps the existing default fallback in both modes, since FSST cannot be used as a general buffer compressor. ## Compatibility No change to stable-format output. Only the unstable 2.3 format can now emit LZ4 per-value pages with nulls, and those require a reader with this fix. Once 2.3 is stabilized, its minimum reader will already include the fix, so LZ4 can reach default users without breaking released readers. For stable formats I kept the existing fallback rather than rejecting the `lz4` selection, because rejecting would turn writes that succeed today into errors for users who already set `lz4` on these columns.
…eto the first (lance-format#9895) ## Summary A vector column can carry more than one index (e.g. an L2 index and a cosine index). When a nearest-neighbour query specified a `metric_type`, the scanner picked the **first** index keyed on that column and then used the query metric only to **veto** it. On a mismatch it warned and brute-forced the whole column — even when a second index on the same column had exactly the requested metric. This makes the metric part of index **selection** instead of a post-hoc veto, so a cosine query on a column holding both `idx_l2` and `idx_cos` now uses `idx_cos`. Fixes lance-format#9873. ## Root cause In `create_plan`'s vector-index branch (`rust/lance/src/dataset/scanner.rs`), the candidate was chosen with ```rust indices.iter().find(|i| i.fields.first() == Some(&column_id) && index_type_is_known(i)) ``` The predicate filters on the keyed column and index type but **not** the metric. The query's `metric_type` was applied only after selection, to compute `use_this_index`; a mismatch set it to `false` and the whole branch returned `None` (brute force). The iterator never advanced to a second index on the same column whose metric did match. Results stayed correct — the query just ran far slower than it needed to. ## Fix Select over **all** indices keyed on the column with a known vector type: - Query specifies no metric → the first candidate answers, exactly as before. - Query specifies a metric → the first candidate whose metric equals it answers. - No candidate matches → the existing warn-and-brute-force path. The metric is resolved through the existing fast path (`metric_type_from_index_metadata`), so a modern index needs no I/O to select; only legacy candidates without details are opened, and the loop breaks on the first match. The segment-loading and coverage checks after selection are unchanged. ## Tests Added `test_knn_selects_index_matching_requested_metric`: build two IVF_PQ indices on the same `vec` column (`idx_l2` with L2, `idx_cos` with cosine), then assert via `explain_plan` that - a **cosine** query hits `ANNSubIndex` with the cosine metric (index used) rather than falling back to `KNNVectorDistance` (brute force), and - an **L2** query still selects `idx_l2`. Without the fix the cosine query brute-forces and the first assertion fails. The existing `test_knn_metric_mismatch_falls_back_to_flat_search` (single index, mismatched metric → brute force) and `test_knn_no_metric_uses_index_metric` (no metric → index's own metric) still pass, since both behaviours are preserved. Also refactored the `TestVectorDataset` helper so `make_vector_index_with_metric` delegates to a new `make_named_vector_index_with_metric(name, metric)`, letting a test place several differently-named indices on one column. ## Scope Purely a selection/perf fix — no change to what any query returns. Only the scanner's index-selection branch and a test helper are touched. Signed-off-by: hello-peter-tang <132056702+hello-peter-tang@users.noreply.github.com>
…mat#9893) ## Summary - Declare `json` and `from_json` on `CompactionPlan`, `CompactionTask`, and `RewriteResult` so pyright no longer reports `reportAttributeAccessIssue` for these existing native methods. - Correct the stub property to `RewriteResult.original_fragments` to match the native getter. - Add a compaction typing regression to the pyright target and align the Rust `RewriteResult.from_json` docstring with the stub. JSON stability warnings are preserved. - Resolve `CompactionOptions` through a relative import so pyright can check the stub before the native extension is installed. ## Test plan - [x] Full configured pyright target passes with pyright 1.1.414 in an isolated environment without pylance: 0 errors, 14 warnings. - [x] Incorrect `target_rows_per_fragment` argument types are still rejected. - [x] `uv run --no-sync make lint-python`: Ruff passes; pyright reports 0 errors and 15 dependency-stub/module-source warnings. - [x] `uv run --no-sync pytest python/tests/test_optimize_typing.py python/tests/test_optimize.py -k 'compaction_json_types or dataset_distributed_optimize' --durations=30`: 8 passed. The new test case takes 0.05 seconds. - [x] Removing the six JSON method declarations and reverting the property name produces 7 pyright attribute errors; the stub was restored after this regression guard check. - [x] The runtime `RewriteResult.from_json` docstring matches the stub. - [x] `cargo fmt --all` passes for the workspace and Python crate. - [ ] Rust clippy was not run locally because the `no-local-cargo-builds` skill prohibits local Cargo lint builds. --------- Co-authored-by: zhanghaobo@kanzhun.com <zhanghaobo@kanzhun.com>
…lance-format#9827) Deleting a nonempty Lance branch on a TOS bucket with hierarchical namespace can fail with HTTP 409 `CannotDelete`: the OpenDAL bridge exposes directory entries as zero-size objects, and the generic deletion stream may remove a directory before its children. The generic `ObjectStore::remove_dir_all` now defers zero-size entries and deletes them longest-path-first after nonempty files. Zero-byte files are deferred too, which is safe because child paths are longer than their parent directory. Nonempty files are forwarded through one continuous `delete_stream`, retaining each provider's batching and concurrency. All file deletes complete before deferred entries are deleted. Cleanup uses a single wrapped listing, so tracing, metrics, filtering, and path rewriting remain in effect; no native OpenDAL listing or provider directory-operation refactor is needed. Regression tests cover hierarchical and flat TOS listings, pagination with each page read once, mixed nonempty and zero-byte files, nested/empty/missing directories, encoded paths, sibling-prefix isolation, deletion failures, rewritten paths, wrapped stores, and dynamic credentials. A synthetic S3-style backend test verifies that 20,000 nonempty files retain 20 concurrent batches and that deferred entries are deleted only after all file deletes finish. Validation: - `cargo test -p lance-io --features tos object_store::`: 277 passed. - `cargo fmt --all` and `git diff --check`. - `cargo clippy -p lance-io --features tos --tests --benches -- -D warnings`. - `cargo clippy --all --tests --benches -- -D warnings` (Rust 1.98.1). - Reproduced the original failure on upstream `4323cdee1374bfe28971a83f72dddf43f9cbc863` (14.0.0-beta.8). Revalidated the revised fix on a disposable real TOS HNS dataset with two main fragments / four rows and a branch appended to six rows: branch deletion succeeds, its files and directories are removed, and the main dataset still reads all four rows. The disposable dataset was fully cleaned up. This fixes deletion ordering; it does not add rollback of an already removed branch reference if cleanup fails. CI status: - [Lance Gatekeeper recommends approve without changes](lance-format#9827 (review)); the empty-commit revision preserves that review. - The earlier transitive `unsafe-libyaml` advisory blocker was addressed upstream in [lance-format#9896](lance-format#9896). The new CI run on `b31ed5953` passes the dependency audit and license-header check. This PR adds no advisory exception. - See the PR checks for the latest build and test results. --------- Co-authored-by: wangzheyan <wangzheyan@bytedance.com>
… in accelerated IVF_PQ (lance-format#9755) Fixes lance-format#9754 `create_index(..., index_type="IVF_PQ", metric="cosine", accelerator=...)` built indices with much lower recall than the CPU build (for example 0.387 vs 1.000 on 500k x 2048-d embeddings with CUDA). The torch path trains the PQ codebook and computes the PQ codes in Python, and the Rust builder uses them unchanged. For cosine the two sides disagreed: - Rust normalizes the vectors, takes residuals of the normalized vectors, and treats the PQ codebook as an L2 codebook over them. - The torch path trained the PQ codebook with cosine k-means, which normalizes codewords and residual sub-vectors to unit length and so drops the residual norms. It also took residuals of the unnormalized vectors. ## Changes All in `python/python/lance/vector.py`, cosine only: - `train_pq_codebook_on_accelerator`: train the per-sub-vector k-means with `l2`. PQ encoding in `one_pass_assign_ivf_pq_on_accelerator` goes through the same `KMeans` objects, so it uses L2 as well. - `compute_partitions` (residual sample for PQ training) and `one_pass_assign_ivf_pq_on_accelerator` (final encoding): normalize the vectors before subtracting the IVF centroid, as the Rust build does. `l2` and `dot` are unchanged. Not changed here: cosine IVF centroids from the torch k-means are still normalized to unit length (lance-format#1759), while the CPU build keeps the mean of the normalized vectors. Encoding and search both use the stored centroids, so the index is consistent, but the residuals are larger. This explains the remaining PQ-only gap in the table below: on the pre-normalized 50k data, PQ-only recall@100 is 0.765 with this PR and 0.832 when the torch k-means also keeps unnormalized cosine centroids (CPU build 0.827). With refine_factor=10 every variant is at 1.000. The centroid change is in `lance.torch.kmeans.KMeans`, which is public and also used by `IndicesBuilder`, so I left it for a follow-up. ## Tests Added `test_torch_cosine_ivf_pq_recall` to `python/python/tests/test_vector_index.py`, next to the other torch-path index tests. It builds a cosine IVF_PQ index on 4096 x 32-d vectors with varying norms, once without accelerator and once with `accelerator=torch.device("cpu")`, and checks that the recall@10 of the accelerated build is within 0.1 of the CPU build. It needs only torch (`pytest.importorskip("torch")`), no GPU, and runs in about 3 s. - Without the fix: CPU 0.840, torch 0.179 (fails). - With the fix, 15 runs: torch recall 0.821 to 0.847, CPU recall 0.834 to 0.864, largest gap 0.035. All Python numbers here come from the released pylance 11.0.0 wheel with this `vector.py` change applied (the changed functions are identical on `main`). I have not built `main` from source. On that setup the torch-related tests of the 11.0.0 test suite (`test_vector_index.py`, `test_indices.py`, `test_f16.py`, `torch_tests/`, filtered with `-k "torch or accelerator or f16 or centroid or cuda or gpu"`) give 49 passed, 26 skipped, 1 failed both with and without the change. The failure is `torch_tests/test_data.py::test_bf16_to_tensor_clones_when_nulls_present`, which does not use `vector.py`. ## Recall before / after recall@100 against exact cosine top-100, 50k x 512-d synthetic data, 64 partitions, 64 sub-vectors, `accelerator="mps"`: | Variant | Before: refine 10, nprobes 32 | Before: PQ only, all partitions | After: refine 10, nprobes 32 | After: PQ only, all partitions | |---|---|---|---|---| | CPU build, cosine | 1.000 | 0.835 | 1.000 | 0.827 | | MPS, cosine, raw vectors | 1.000 | 0.502 | 1.000 | 0.768 | | MPS, cosine, pre-normalized vectors | 0.690 | 0.110 | 1.000 | 0.765 | | MPS, l2, pre-normalized vectors (reference) | 1.000 | 0.834 | 1.000 | 0.833 | recall@10, 4096 x 32-d (script in the issue), `accelerator=torch.device("cpu")`: | Data | Before: CPU build | Before: torch build | After: CPU build | After: torch build | |---|---|---|---|---| | raw vectors | 0.839 | 0.188 | 0.857 | 0.827 | | pre-normalized vectors | 0.846 | 0.269 | 0.844 | 0.838 | The CPU build is not seeded, so its numbers vary by about 0.02 between runs. Co-authored-by: Chastrlove <17813559+Chastrlove@users.noreply.github.com>
…ce-format#9701) Description The type stubs for the MemWAL classes in `lance/lance/__init__.pyi` had drifted from the PyO3 signatures, so pyright both rejected valid calls and accepted invalid ones. | Method | Stub before | Binding | |---|---|---| | `_LsmScanner.limit` | `n` required | `n` defaults to `None`, so `limit(offset=10)` is valid | | `_LsmVectorSearchPlanner.plan_search` | no `refine_base_table` / `overfetch_factor` | both supported | | `_LsmVectorSearchPlanner.__init__` | no `filter` | `filter` supported | | `_ShardWriter.lsm_scanner` | accepts `None` | takes a list; `None` raises `TypeError` at runtime | `lance/mem_wal.py` is also added to pyright's `include` list, so future drift in the first two kinds is caught in CI. The last two are not visible through that file: it passes `filter` via `**kwargs` and never passes `None` to `lsm_scanner`. No runtime behavior changes.
…e-format#9693) `LsmScanner::try_into_stream` returned JSON columns as raw JSONB (`large_binary`), so Python `to_table` / `to_batches` disagreed with `Dataset.to_table` and Java `scanBatches()`, which return Arrow JSON. **Reading the same rows through different APIs gave different results.** So convert in `try_into_stream`, as `Scanner::try_into_stream` does. `count_rows` skips the conversion.
…ance-format#9641) `execute_plan` and `analyze_plan` copy `LanceExecutionOptions::batch_size` straight into the session config with `options_mut()`, which bypasses the `batch_size > 0` assertion in `SessionConfig::with_batch_size`. A batch size of 0 then reaches DataFusion's operators: executing a `SortExec` with it panics with `attempt to divide by zero` in `datafusion-physical-plan` (`sorts/stream.rs:379`, DataFusion 54.1). Return an `InvalidInput` error instead, before any plan runs. Dataset scans already validate their batch size (`validate_batch_size`), so only direct callers of `execute_plan` / `analyze_plan` / `analyze_plan_with_context` are affected. `get_task_context` is private, so no public signature changes. This is also what DataFusion 55 requires: its `execution.batch_size` becomes a `ConfigNonZeroUsize`, so the upgrade in lance-format#8997 only has to swap the check for `ConfigNonZeroUsize::try_new`. Added `test_zero_batch_size_is_rejected`, which checks both entry points for the error variant and message. Ran `cargo fmt --all`, `cargo clippy -p lance-datafusion --tests -- -D warnings`, and `cargo test -p lance-datafusion --lib -- batch_size`. Part of lance-format#9647. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ce-format#9639) ## Summary `Scanner::nearest` used the entire child array when flattening a `ListArray` query. For sliced lists, this included values outside the visible slice, causing incorrect query dimensions or extra multivector query inputs. Slice the child array using the first and last visible offsets, preserving zero-copy behavior and the existing query-count semantics. ## Validation Regression tests compare sliced and equivalent list queries for flat and indexed search across multiple fragments, including prefix, middle, suffix, and unsliced cases. They also check multivector query keys and fixed-size-list query handling. - `cargo test -p lance --lib test_sliced_list_` — 2 passed. - `cargo fmt --all` — passed; no changes. - `cargo clippy --all --tests --benches -- -D warnings` — passed.
…ance-format#9637) ## Summary When `n > max_takes` and no filter is given, `maybe_sample` takes `n // max_takes` rows from each chunk. If `n` is not a multiple of `max_takes` it returns fewer rows than requested (on a 10,000-row dataset with the default `max_takes=2048`, asking for 3000 or 4095 rows returns 2500). If `n` is a multiple, it stops after `max_takes` chunks and never samples the tail of the dataset. This is the path used by `lance.torch.data.LanceDataset(samples=N)`. The fix samples each chunk in proportion to its size, so the counts add up to exactly `n` and cover the whole dataset, and rounds the chunk size up so the number of takes stays within `max_takes`. ## Test Added `test_sample_dataset_returns_n_rows` to `python/python/tests/test_sampler.py` (n = 100, 3000, 4095). It fails on main (99, 2992, 4080 rows) and passes with the fix. `ruff format --check` and `ruff check` pass on the changed files; pyright was not run locally.
## Summary `ShardedFragmentSampler` and the unfiltered path of `ShardedBatchSampler` seed their shuffle with `seed` alone, so with `randomize=True` every epoch yields the same order and `set_epoch()` has no effect. `ShardedFixedBatchSampler` and the filtered `ShardedBatchSampler` path already seed with `seed + epoch`; this makes the other two do the same. Epoch 0 keeps its current order, and since every rank uses the same `seed + epoch`, shards stay disjoint. ## Test Added `test_set_epoch_reshuffles` to `python/python/tests/test_sampler.py`, parametrized over both samplers. It fails on main (identical order for epochs 1 and 2) and passes with the fix. `ruff format --check` and `ruff check` pass on the changed files; pyright was not run locally.
…rmat#9556) `compact_files_with_planner` runs the rewrite tasks through `buffer_unordered(num_threads)`. With `num_threads` set to `Some(0)` that stream never pulls a task and returns `Pending` with nothing left to wake it, so compaction neither finishes nor fails; in Java it also keeps the dataset's write lock, so `close()` blocks too. The value arrives unchanged from Python and Java, and zero often means "pick automatically" elsewhere (rayon treats it that way). Reject it once the plan is built, before any file is rewritten, which also covers plans from a custom `CompactionPlanner`; `plan_compaction` does not use the value and is unchanged. It is rejected rather than replaced by the default, like the zero-parallelism guard lance-format#7632 added to filtered reads. With nothing to compact, `Some(0)` now returns an error instead of `Ok`. From Java the error surfaces as `IllegalArgumentException` from `compact()`. ## Testing `cargo test --release -p lance test_compact_files_rejects_zero_threads`, which times out on the old code, plus fmt and clippy with `-D warnings`.
…ance-format#9552) `DatasetDeltaBuilder::build` rejects `compared_against_version` combined with a begin/end range, but both bindings picked one mode before the core saw them. Python's `LanceDataset.delta()` forwarded the range only when `compared_against` was `None`, and the JNI `nativeBuild` forwarded it only when `comparedAgainst` was unset. So `ds.delta(1, begin_version=3, end_version=7)`, or the same selection on the Java `DatasetDeltaBuilder`, returned the delta between version 1 and the dataset's current version with no error, although the Python docstring promises a `ValueError`. Both bindings now forward every argument and let `build` reject the combination. ## Behaviour change Combining the two modes raises `ValueError` in Python and `IllegalArgumentException` in Java instead of returning a delta. In Java, setting only one end of the range still raises `IllegalArgumentException`, now with the core's "Must specify both" message instead of "Must specify either". `Dataset.delta(long)` and `Dataset.delta(long, long)` set a single mode and are unaffected. ## Tests `test_delta_validation_errors` and `DeltaTest` gain the conflicting case, which returned a delta before the fix; `DeltaTest` also pins the half-set message. Ran `pytest python/tests/test_delta.py` and `./mvnw test -Dtest=DeltaTest`, plus `make lint-python`, spotless, and clippy with `-D warnings` on `lance-jni`.
…-format#8479) Adds V2.0-V2.3 regression coverage for metadata-only all-null column addition followed by compaction, including ForceBinaryCopy rejection, safe re-encode fallback, row/order/null preservation, and persisted physical mapping. Also corrects the Python BITMAP/LABEL_LIST row-address compatibility boundary so 15.0.0b2 is treated as a supporting reader while earlier versions retain full-scan fallback coverage.
…e-format#10350) Streaming coreset training returned Float32 centroids for Float16 vector columns. A complete IVF_RQ build then failed when residual computation received those centroids alongside Float16 vectors. Convert the final centroids back to Float16 after optional raw-vector refinement, keeping training in Float32 and aligning the model with the column for index construction and queries. The regression builds one-bit cosine IVF_RQ indexes with 257 partitions over two fragments, reopens the dataset, verifies the persisted centroid type and index query plan, and asserts recall against brute-force search. It probes all partitions and reranks the full 600-row fixture so random one-bit quantization cannot discard true neighbors before refinement. Cases cover both streaming samplers, omitted refinement, Float32 streaming, and Float16 in-memory training. The fixed-sampler Float16 cases reproduced the exact reported type mismatch before the fix; both controls passed. Validation: - Five build-and-query regression cases pass in 20 consecutive runs (100 cases). - `cargo test --profile ci -p lance --lib index::vector::ivf::tests::`: 123 passed. - `cargo clippy --all --tests --benches --profile ci -- -D warnings` passes. - `cargo fmt --all -- --check` and `git diff --check` pass. Fixes lance-format#10349 <!-- lance-gatekeeper-fix:v1 agent=af546cbd4568cbfd3cd4deb11866b7c4 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…at#8400) Preserve compaction row order across parallel, partial, and gapped rewrites while keeping manifests sorted by fragment ID. Compaction results commit as one ordered replacement suffix. Metadata-only trailing relabels reserve fresh IDs, copy deletion files into the compacting dataset (including shallow clones), remap physical-address indexes, and retain the released table format. Row-adding concurrent writes still require replanning, while stale tasks can rebase across prior metadata-only relabels. The strict `max_source_fragments` bound remains a documented, recoverable limitation tracked by lance-format#8986. Fixes lance-format#3465
Bound indexed merge-insert execution with disk-backed source replay, capped spillable sort inputs, and a spillable sort-merge join. Preserve provider snapshots, `FirstSeen` encounter order, partial-index fallback behavior, and stable row IDs across write batches and output fragments. The accepted limits remain: low-cardinality index candidate maps do not spill, pipeline RSS is not a hard scan-wide cap, staging adds temporary I/O, and throughput is unbenchmarked. Fixes lance-format#8129
…at#10268) Bounds legacy take concatenation before Arrow offset capacity is exceeded, preserves duplicate rows, ordering, and carried-column alignment, and adds regression coverage for legacy row groups and indexed updates.
Fragment-scoped vector search now accepts the default postfilter path, combines selected ANN coverage with flat fallback, and constrains candidates before top-k. The change includes regression coverage for postfilters, fast search, deletions, overlays, stable row IDs, and fragment-reuse mappings.
Automatically batch reads of overflowed dictionaries under top-level primitive and struct decoders while preserving the recovery and exact page-merging semantics now on main. The change adds capacity-aware batch limits for direct primitive and struct dictionary decoders and keeps stable historical regression coverage for Lance 2.0 and 2.1 plus pylance 10.0.0 and 13.0.0. Dictionaries nested under lists remain outside the automatic batching scope and may require an explicit smaller read batch. Decoder recovery and page concatenation retain `NotSupported` capacity errors.
## Why ZoneMap V0 uses `max = +NaN` to mark any zone containing NaNs, including negative NaNs. This hides the ordinary maximum and does not tell the reader which NaN signs are present. lance-format#10102 added ordinary bounds and separate negative-NaN counts to write seeds. This PR brings the same information into **ZoneMap index V1**, implementing the format proposed in lance-format#9986. It does not change the seed format again. ## What V1 stores For example, a zone containing `[1.0, 5.0, -NaN, +NaN]` is represented as: | Field | V0 | V1 | | --- | --- | --- | | `min` | `1.0` | `1.0` | | `max` | `+NaN` marker | `5.0` | | `nan_count` | `2` | `2` | | `negative_nan_count` | Not stored | `1` | V1 keeps these meanings separate: - `min` / `max`: bounds on non-null, non-NaN values, following **Arrow Rust ordering**. Infinities are included. Bounds are null when there are no ordinary values; nested types retain their existing null-only behavior. - `null_count`: number of nulls. - `nan_count`: number of all NaNs. - `negative_nan_count`: number of those NaNs with the sign bit set. Positive NaNs are `nan_count - negative_nan_count`. The reader uses the bounds and NaN counts together for conservative pruning. NaNs no longer overwrite an ordinary bound in a V1 file. ## What happens to existing indexes? **New indexes use V1; existing V0 segments remain readable and are not eagerly rewritten.** Both versions can serve the same logical index. Each segment is decoded according to its own `index_version`. | Operation | Behavior | | --- | --- | | Create a new segment | Write V1 | | Dataset maintenance selects a V0 segment for replacement | Rebuild its covered data as V1 | | Low-level `ScalarIndex::update` | Preserve the input version, following the existing convention | | Direct merge of V0 + V0 | Write V0 | | Direct merge of V1 + V1 | Write V1 | | Direct merge of V0 + V1 | Reject before writing; rebuilding is required | A valid V0 seed is eligible for V1 reuse only when its stored `nan_count` is explicitly zero; its negative count is then provably zero. Missing counts are not zero. A V0 seed containing NaNs cannot supply V1's ordinary maximum and exact sign split, so the update falls back to scanning rather than inventing statistics. Every output V1 record must satisfy the V1 contract; conservative V0 query compatibility does not relax it. This PR retains coarse-grained fallback; finer-grained seed reuse is separate work. Optional V0 seeds with invalid counts or incomplete fragment coverage also fall back to scanning before reaching the strict V1 writer. ## Safety and compatibility - V1 read/write validation rejects impossible counts and NaN bounds. A retained, shortened FRI-remapped zone is also rejected by direct merge when its counts can no longer be established exactly; callers must rebuild it instead. Destination fragments excluded from the merge are filtered before fallible remap-run construction and validation. - The existing `index_version` filter lets readers without V1 support skip V1 segments and fall back to scans. New readers continue to read V0 segments under their original contract. - No new table feature flag is added. Zone sizing, configurable null-bitmap collection, and packed in-memory storage are outside this PR. ## Tests Coverage includes V0/V1 round trips, invalid statistics, unknown versions, seed reuse/fallback, merge-version rules, remap-count safety, a released V8 fixture rebuilt as V1, and fresh V1 segments committed alongside untouched V0 coverage. Dataset tests assert the index is used and results equal a scan. Validation at `f497c6c97`: - GitHub checks completed with no failures, including Linux x86/ARM, macOS, Windows and compatibility tests. The external-plugin test fixture now advertises the ZoneMap version it delegates to, fixing the earlier four-platform CI failure. - Author-run fmt and clippy passed; `lance-index` 2,775 passed / 3 ignored. Targeted ZoneMap tests: 68 lance-index and 117 Lance tests passed; external-plugin integration tests: 7 passed. - Author-run Lance library: 6,402 passed / 1 failed / 4 ignored. The failure was the FTS background tiered-merge timing test; all five isolated reruns passed. That local full invocation was **not entirely green**. Other integration suites and doc tests passed. - Author rebuilt Python bindings on the same head: targeted tests 25 passed; compatibility 7 passed / 1 skipped (the suite excludes ZoneMap 0.x downgrade); Python lint passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Pin the pip-installed Python test extra to pytest 8.4.2 and pytest-xdist 3.8.0, matching the existing uv test group. The macOS CI jobs resolved pytest 9.2.0 before this change. With that runner, the seven lindera fixture tests were distributed across workers and raced on the shared extracted dictionary. The corrected head installs pytest 8.4.2, serializes all seven tests with `@lindera`, and passes the complete Python workflow (1,895 passed, 530 skipped on macOS).
Record the vector-per-partition target implied by an explicit partition count or centroids for full-table IVF builds. The target is derived from the build's actual shuffled vector count and effective trained partition count, then carried through index details, append, optimize, and retrain. Explicit positive targets still win; fragment subsets, multivector builds, and zero-vector builds retain the default; explicit zero targets are rejected with or without a partition count or centroids. The new public `VectorIndexBuildSummary::num_vectors_shuffled` field requires the breaking marker. Validation: current-head Rust, Python, Java, module-size, license, typo, format gate, and repository checks passed after merging the latest main and the compatible pytest dependency pin. The Python macOS suite passed 1,895 tests with 530 skipped. The targeted 13 Rust tests and three Python retrain cases passed at 484798b; Gatekeeper approved with the documented retrain risk and all threads are resolved.
…e-format#10387) ## Summary The quickstart only showed PyLance, but most users are better served by a fuller engine. This adds LanceDB and Apache Spark quickstarts next to PyLance, modeled on how the Iceberg site offers one quickstart per engine. LanceDB becomes the landing page, so the site's "Get started" buttons now go there. ```text Getting Started ├── LanceDB quickstart/ (new; was the PyLance intro) ├── Apache Spark quickstart/spark/ (new; SQL + PySpark tabs) └── PyLance ├── Getting Started quickstart/pylance/ (moved from quickstart/) ├── Versioning (URLs unchanged) ├── Vector Search └── Full-Text Search ``` Both new pages walk through the same product catalog: create a table, filter it, run vector and full-text search, and modify it. LanceDB adds time travel. Spark ends by opening the table it wrote from LanceDB, to show that the engines share one format. The page code is not inline Markdown. It is embedded with `pymdownx.snippets` from runnable files in `docs/snippets/quickstart/`, and each step's output is shown below its code. The output comes from files in `outputs/` that a new `docs-snippets` workflow checks against a fresh run: - Python snippets run section by section like notebook cells. Each section's printed output and the value of its final expression are recorded. - The Spark SQL file runs through the exact `spark-sql` command the page shows. Marker queries split its output by section. - If the output drifts, CI fails with a diff. `pytest --update-outputs` re-records it. - Vectors use multiples of 0.125 so distances are exact in float32, and pandas output is rounded to 4 decimals, so recorded output doesn't vary by platform. The workflow also checks that the `pip install` commands on the pages accept the versions under test. `check_paths: true` makes the docs build fail on a broken snippet reference instead of rendering an empty block. Pins and caching: - Python deps are pinned in `docs/snippets/pyproject.toml` and `uv.lock`, and cached by `setup-uv`. I added a `!snippets/uv.lock` exception to `docs/.gitignore`, which otherwise ignored the lockfile. - The Spark connector bundle (`lance-spark-bundle-4.2_2.13:0.8.0`) pulls about 577 MB of jars. They are cached under a key hashed from the files that name the coordinate, with a prefix fallback so a version bump downloads only what changed. - Dependabot bumps the Python pins monthly, in one grouped PR with a 2-day cooldown. The connector coordinate is bumped by hand: a PySpark minor bump fails CI until it is updated, and that failure serves as the reminder. ## Evidence Locally (JDK 17, Python 3.12), all five snippet tests pass in about 30s with warm jars. The full-site build, run with lance-spark's docs assembled in, shows no new warnings. The links from `spark.md` to `../integrations/spark/...` only resolve in that assembled build, like the other integration links. ## Merge Risk **Door:** two-way. Docs, CI, and Dependabot config only. The one URL change: `quickstart/` now shows LanceDB instead of the PyLance intro, which moved to `quickstart/pylance/`. **Blast Radius:** docs site, CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
UpdateBuilder::set rejected any column containing blob v2 values, so an UPDATE could carry blob columns through unchanged but never assign one. A column holding blob v2 values at the top level or inside structs, lists or large lists now takes an expression that yields bytes in their place: the bytes are wrapped in the logical blob layout before the write path, which stores them inline, packed or dedicated as usual. The rewrite feeds the assigned blob values in as logical placeholders, found from the dataset schema, so it no longer reads old blobs that the assignment overwrites, and it keeps the other fields of a struct. An update that reads a blob column it also assigns is rejected. Blob v2 values inside maps and fixed-size lists stay unsupported: Lance cannot write the former, and its file reader cannot read a fixed-size list of structs back.
…ow (lance-format#10339) Constant pages (2.2+) were decoded one row at a time. `skip_to_row` stepped through every row before the requested one, and `drain` called `take_row` and recounted visibility once per row. Each take builds a new decoder whose cursor starts at the head of the page, so reading a single row near the end of a page walked the whole page. Without repetition levels every row is exactly one level, so jump the cursor straight to the target row, as the complex all-null decoder already does. Each drained range is then one level slice. `drain` only counts visible items when there are no definition levels; when there are, `decode` already counts them from the sliced levels. Pages with lists still step row by row to find the range ends. The decoded values are now wrapped with `DataBlock::from_decoded_array`, which skips encoder statistics and reuses the fixed-width buffer instead of copying it. Output is unchanged. The existing constant-layout tests now also read ranges and takes that start mid-page. **Measured on a 2.2 file with a single-value int64 column, pinned to 16 cores, best of 5**, two interleaved rounds: | Case | Before | After | | --- | ---: | ---: | | scan 4M rows | 13.4-14.3 ms | 3.5-3.6 ms | | scan 1M rows | 2.4-2.6 GB/s | 16.4-17.8 GB/s | | take 100 rows out of 1M | 2.05 ms | 0.01 ms |
…t#10393) lance-format#9977 fixed DynamoDB key-type validation and paginated manifest deletion for lance-format#9974, which is now closed. This test-only follow-up preserves the three cases that were unique to duplicate PR lance-format#9980: - A 26-row query page is split into exact 25-key and 1-key `BatchWriteItem` requests, with the full request sequence and key sets asserted. - Continuously returned `UnprocessedItems` are re-sent until the production bound is exhausted. The test asserts the initial send plus eight re-sends, the explicit `Error::IO` and diagnostic text, and that later requests contain only the remaining key. - Query responses with missing `Items` and with `Items: []` both complete successfully without issuing `BatchWriteItem`. The tests use the existing scripted endpoint and real AWS SDK path; no production code or retry configuration changes are included. Dataset deletion automatically cleaning DynamoDB manifest rows remains lance-format#3494 and is outside this change. The retry-exhaustion test drives all nine batch requests through paused Tokio time and the real SDK to a local scripted endpoint. It completed in 0.13s without hitting an SDK network timeout.
Closes lance-format#9647. DataFusion 55 and Arrow 59 require a coordinated upgrade across the Rust workspace, Python bindings, and Java JNI crate. This PR carries the original implementation forward, resolves its conflicts with the current `main`, and adapts the remaining execution-plan, spill, and binding APIs to DataFusion 55. This replaces lance-format#9979 because that branch is owned by an organization where the maintainer cannot push. The original commit authors are preserved in the branch history; the trailers at the end of this description also retain their credit if the PR is squash-merged. ### Behavior changes - Migrate execution-plan expression reporting, statistics, planning contexts, optimizer composition, and `AnalyzeExec` construction to DataFusion 55. - Preserve full-text-search ordering during the `EnforceDistribution` removal, align indexed and hash-join signed-zero matching, and keep Arrow 59 map copying behavior compatible with existing data. - Use DataFusion 55's public `SpillFile` API and refresh the Rust, Python, Java, and Python dependency lockfiles. - Package the local Arrow helper crates in CI while their `59.0.0` versions are not yet on crates.io. ### Verification - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo check --workspace --tests --benches`, including `--all-features` - `cargo package --workspace --no-verify --all-features` - Targeted Rust tests for DataFusion, spill handling, merge-insert behavior, Arrow ordering, count pushdown, top-k late materialization, full-text search, and execution plans - Python `make install`, `uv run make lint`, and `test_table_provider.py` - Java JNI clippy, Spotless, and `./mvnw test` Thanks to @timsaucer for the original implementation and careful behavior analysis, and to @slachiewicz, @wjones127 for the review, signed-zero discussion, and follow-up coverage. Their contributions and review history made this upgrade possible. Co-authored-by: Sylwester Lachiewicz <slachiewicz@apache.org> Co-authored-by: Tim Saucer <timsaucer@gmail.com> --------- Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Sylwester Lachiewicz <slachiewicz@apache.org>
…concat (lance-format#9912) Preserve add-column stream sub-batches through the updater so large variable-width outputs do not require a 32-bit-offset concat. Deleted-row alignment and legacy row-group behavior remain unchanged.
…9869) Delete data, deletion, and owned managed Blob payloads written by merge_insert analyze plans without committing, while preserving existing fragments and reused Blob references. Closes lance-format#9862.
…ance-format#10406) Wait for asynchronous replay-spill cleanup before asserting quota release This is a test-only synchronization fix for the intermittent DataFusion spill quota check. It preserves the assertion that disk reservations must eventually reach zero while allowing the background drain task to finish cleanup; library behavior is unchanged.
…mespace (lance-format#9867) Implement DirectoryNamespace create/drop mode semantics and require existing parent namespaces before table creation or declaration. The final integration preserves namespace-specific lookup, object-collision checks before marker writes, canonical table locations, and root-equivalent drop protection. Destructive overwrite and cascade modes remain explicitly unsupported.
…rmat#10413) Nullable struct merges can panic on a sliced Boolean child because the rebuilt null bitmap does not cover the child array offset. Align the merged validity bitmap with the child offset before rebuilding the array, preserving the existing buffer when the offsets already match. Add 84 regression combinations over child/parent bit offsets 0, 1, 7, 8, 9 and 1025, with and without child nulls. The expected logical values are independent of the bitmap representation. This completes the sliced-child case left uncovered by lance-format#9503; related downstream issue: [LanceDB lance-format#4260](lancedb/lancedb#4260). Validation: - Before the fix: 35 of the 84 new cases fail on main. - `cargo test -p lance-arrow`: 203 unit tests and 6 doctests pass after the fix. - `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D warnings` pass. The equivalent Arrow 58 backport for the planned 14.0.2 patch release is lance-format#10412. It is being validated against freshly fetched LanceDB main, without the downstream vendor override.
…rmat#9769) Record metadata-only suffix relabels with all moved physical row addresses, including deleted rows. Integrate strict trailing-suffix planning with recorded and unrecorded compaction races, and cover the resulting Rust and Python paths. Keep the accepted lance-format#9947 follow-up outside this PR scope.
…mat#9721) ## Summary - Follow-up to lance-format#9518 (merged). Addresses the non-blocking risk the gatekeeper flagged there: `JsonIndexPlugin::load_index` passed `target_plugin.version()` (the target plugin's current max format version) instead of the version the target index was actually persisted at, since `JsonIndexDetails` never recorded it. - Adds an optional `target_index_version` field (field 4) to `JsonIndexDetails`, populated from `CreatedIndex::index_version` whenever the wrapper trains, remaps, or updates its target. `load_index` now reads it, falling back to the target plugin's current version only for details written before this field existed (matching the previous, safe-for-now behavior for old data). - Drops the now-stale note in the `target_data_type` docs saying these details do not record the target's format version. - This matters for the stacked [BTree format change](lance-format#9481), which will use the passed-in version to distinguish stored row IDs from row addresses. ## Test plan - `cargo clippy -p lance-index -p lance --tests -- -D warnings` clean - `cargo fmt --all -- --check` clean - `cargo test -p lance-index --lib scalar::json::` — 27 passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018BsnoP9ZPMX6vhJSygzdnN --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…10417) Follow up on [the review comment in lance-format#10413](lance-format#10413 (comment)). A typed `NullBuffer` preserves its own bit offset. Arrow 59.3 validates its backing byte length against `ceil((child.offset() + child.len()) / 8)`, so different offsets do not always require bitmap realignment. Reuse a buffer that already meets that bound and retain the existing padding fallback for an undersized buffer. Add four cases covering offsets within one byte, different offsets, and small slices with large child or parent offsets. They verify logical null values, shared backing storage, full ArrayData validation and IPC round trips. The four storage-sharing checks fail before this change and pass afterward. Validation: - `cargo test -p lance-arrow`: 207 unit tests and 6 doctests passed, including the existing 84 offset regressions. - `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D warnings` passed. - Buffer reuse is verified through pointer identity; throughput and RSS were not benchmarked. This change targets main's typed-null-buffer path. The validated 14.0.2 RC uses the separate raw-bitmap backport from lance-format#10412.
…ivf-ownership-20261028
Owner
Author
|
Merged upstream in lance-format#9638. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Published upstream as lance-format#9638. This fork PR tracks the same branch.
Summary
Fix an existing stale-vector correctness gap in shared-scan batch IVF search, following up lance-format#7640.
An in-place vector update (
update_columns/Operation::Update) can preserve a fragment's ID and row IDs while pruning that fragment from the old index segment'sfragment_bitmap. Afteroptimize_indices(append)indexes the replacement in another delta, the old index file still physically contains the pre-update vector.ANNIvfBatchExecsearched all deltas with the shared dataset prefilter, so the old segment could return that stale vector. Single-query search already handles segment ownership; this patch reuses its existing helpers:prefilter_for_segment, restricting candidates before local top-k.restrict_to_segmentbefore the cross-delta merge, with the same all-segments-have-bitmaps gate as single-query search.prefilter_for_partitiononce per distinct loaded batch partition, before CPU scoring. Fully owned partitions regainNoFilterand bulk scoring; partitions containing stale entries keep ownership filtering before top-k. This preserves the single-query safeguard from fix(index): filter stale vector segment rows before top-k lance-format/lance#8351, bounded streaming, and load/scoring overlap.No public API, output-schema, or index-format changes. Adaptive probing is a separate follow-up.
Regression coverage
test_batch_knn_skips_rows_owned_by_newer_deltacovers both physical row addresses and stable row IDs:Removing the ownership filtering reproduces the failure in both row-ID cases.
The append-only fast-path test now exercises actual batch partition search with a segment prefilter. A FLAT / 4-bit PQ / 8-bit PQ matrix additionally verifies:
All three new matrix cases failed on the previous PR head because batch search never checked partition coverage; they pass with the shortcut restored. These are deterministic scoring-path assertions, not new timing measurements.
Validation
Validated on upstream base
f4f724131.cargo test -p lance --lib: 4,509 passed, 0 failed, 3 ignored.--all --tests --benches -- -D warnings) and lance all-features clippy: clean.cargo fmt --all -- --check: clean.